[FLINK-38424][planner] Support to parse VECTOR_SEARCH function #27039

fsk119 · 2025-09-25T07:12:09Z

What is the purpose of the change

Support to parse VECTOR_SEARCH function.

Brief change log

Add SqlVectorSearchTableFunction
Add type inference for the newly added function
Set the correct scope for VECTOR_SEARCH function

Verifying this change

This change added tests and can be verified as follows:

Added plan tests for VECTOR_SEARCH function

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

flinkbot · 2025-09-25T07:19:05Z

CI report:

06c61b8 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

...le-common/src/main/java/org/apache/flink/table/connector/source/VectorSearchTableSource.java

davidradl · 2025-09-25T07:36:01Z

...le-common/src/main/java/org/apache/flink/table/connector/source/VectorSearchTableSource.java

+ * vectors during runtime.
+ *
+ * <p>Compared to {@link ScanTableSource}, the source does not have to read the entire table and can
+ * lazily fetch individual values from a (possibly continuously changing) external table when


It would be interesting to compare it to a LookupTableSource also.

I will add this part in the document.

davidradl · 2025-09-25T07:43:47Z

.../main/java/org/apache/flink/table/planner/functions/sql/ml/SqlVectorSearchTableFunction.java

+                                PARAM_SEARCH_TABLE,
+                                PARAM_COLUMN_TO_SEARCH,
+                                PARAM_COLUMN_TO_QUERY,
+                                PARAM_TOP_K));


I an see what TOP_K is from googling. It would be useful to add in the documentation describing the parameters with this change - including the default. I wonder if we should add paging, to be able to handle a large number of results or is this not done with vector databases.

Of course, the document will be added when all tasks are finished. The current API doesn't support paging, I think we can leave this as the future work.

lihaosky

Thanks @fsk119 ! Left some comments

lihaosky · 2025-10-15T23:56:51Z

...able/flink-table-planner/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java

+                            throw new ValidationException(
+                                    "The query column is not literal, please use LATERAL TABLE to run VECTOR_SEARCH.");
+                        }
+                        SqlValidatorScope scope = getSelectScope((SqlSelect) binding.operand(0));


nit: looks there's no need to cast based on line 2618?

It requires. Because line 2618 uses SqlCall to get operand(its return type is <S extends SqlNode> S ), but here we uses SqlCallBinding to get operand(its return type is SqlNode). Therefore, we still need cast here. 0.0

We can not use SqlCall to extract operand because VECTOR_SEARCH allows user to use named argument, which means the operands is out of order and we need to use the name to reorder the operands.

lihaosky · 2025-10-16T00:03:50Z

.../main/java/org/apache/flink/table/planner/functions/sql/ml/SqlVectorSearchTableFunction.java

+    private static final String PARAM_SEARCH_TABLE = "SEARCH_TABLE";
+    private static final String PARAM_COLUMN_TO_SEARCH = "COLUMN_TO_SEARCH";
+    private static final String PARAM_COLUMN_TO_QUERY = "COLUMN_TO_QUERY";
+    private static final String PARAM_TOP_K = "TOP_K";


Missing the optional config param from FLIP?

Yes. But I plan to add this in the https://issues.apache.org/jira/browse/FLINK-38430

lihaosky · 2025-10-16T00:14:55Z

...able/flink-table-planner/src/main/java/org/apache/calcite/sql/validate/SqlValidatorImpl.java

+                                                    Collections.singletonList(SqlKind.SELECT)))) {
+                        boolean queryColumnIsNotLiteral =
+                                binding.operand(2).getKind() != SqlKind.LITERAL;
+                        if (!queryColumnIsNotLiteral && !lateral) {


Isn't lateral always needed? What's the syntax for literal?

LATERAL is not always needed if the SqlCall doesn't contain correlation. For exmaple, users can use the following statement to search.

SELECT * FROM TABLE(VECTOR_SEARCH(TABLE VectorTable, DESCRIPTOR(`g`), ARRAY[1.5, 2.0], 10))

Here, the query input is ARRAY[1.5, 2.0].

CC

flink/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/stream/sql/VectorSearchTableFunctionTest.java

Line 95 in 06c61b8

"SELECT * FROM TABLE(VECTOR_SEARCH(TABLE VectorTable, DESCRIPTOR(`g`), ARRAY[1.5, 2.0], 10))";

I thought it's the syntax but your test expects an exception, so it's not a valid sql?

It's a valid sql. But we don't add literal related rule in physical phase, so planner can not translate the sql correctly. But the exception indicates the planner can parse the statement correctly.

Got it. I guess it's more clear to have something like https://github.com/apache/flink/pull/26553/files#diff-19970e8600e459e820e1310beed925a10450f695698257d85648e8114b5e5aaeR92 to indicate it's not invalid case.

davidradl reviewed Sep 25, 2025

View reviewed changes

...le-common/src/main/java/org/apache/flink/table/connector/source/VectorSearchTableSource.java Outdated Show resolved Hide resolved

davidradl reviewed Sep 25, 2025

View reviewed changes

github-actions bot added the community-reviewed PR has been reviewed by the community. label Sep 25, 2025

fsk119 added 2 commits October 14, 2025 19:07

[FLINK-38424][planner] Support to parse VECTOR_SEARCH function

9db6463

rebase and address comments

06c61b8

fsk119 force-pushed the parse branch from 6209929 to 06c61b8 Compare October 14, 2025 12:51

lihaosky reviewed Oct 16, 2025

View reviewed changes

lihaosky approved these changes Oct 16, 2025

View reviewed changes

fsk119 merged commit 56bf7c8 into apache:master Oct 16, 2025

[FLINK-38424][planner] Support to parse VECTOR_SEARCH function #27039

[FLINK-38424][planner] Support to parse VECTOR_SEARCH function #27039

Conversation

fsk119 commented Sep 25, 2025

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidradl Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lihaosky left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Sep 25, 2025 •

edited

Loading

davidradl Sep 25, 2025 •

edited

Loading